diagonalize the matrix 3x3